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1. 


INTBODUCTIOH 


Hearn  and  Lawphongpanich  (1987)  studied  the  ascent  nature  of 
generalized  linear  programming  (GLP)  which  is  also  known  as  the 
dual  cutting  plane  or  primal  column  generation  technique.  In 
particular,  they  examined  the  properties  of  the  direction,  dot^p , 
defined  as  the  difference  of  two  successive  dual  iterates 
generated  by  GLP  with  respect  to  the  Lagrangian  dual  function,  L. 
They  showed  that  daL.p  is  an  ascent  direction  for  L  at  points 
where  L  is  differentiable.  At  nondi f f erentiable  points,  the 
column  entering  the  master  problem  is  not  unique  and  an  arbitrary 
choice  can  make  d<>i.p  a  nonascent  direction.  To  obtain  an 
additional  insight  on  the  ascent  nature  of  doup ,  they  also  showed 
that  d<»i.p  is  a  solution  to  a  direction  finding  problem  for  L. 

In  this  paper,  we  introduce  a  new  class  of  direction  finding 
problems  and  showed  that  it  includes  the  one  which  produces  da^p . 
This  class  of  problems  contains  direction  finding  problems  which 
always  produce  ascent  directions  as  well  as  those  that  do  not, 
e.g.,  the  dai.p  direction  finding  problem.  In  order  to  improve  the 
rate  of  convergence  of  GLP,  we  replace  the  d®i.p  direction  finding 
problem  with  another  from  this  new  class.  This  new  direction 
finding  problem  which  we  describe  below  guarantees  to  produce 
ascent  directions  and  is  similar  to  the  one  used  by  Demayanov  for 
the  minimax  problem.  Moreover,  a  line  search  step  is  also 
included  in  this  modification  of  GLP.  Hearn  and  Lawphongpanich 
(1987)  indicated  via  a  numerical  example  that  a  line  search  step 
reduces  the  number  of  iterations  for  GLP. 


\ k%»  •'•  >*«  ■*« 


Fop  the  remainder  of  the  paper,  we  state  GLP  algorithm  in 


both  the  cutting  plane  and  the  column  generation  form  and  define 


the  associated  Lagrangian  dual  problem  in  Section  2.  Then, 


Section  3  describes  the  new  class  of  direction  finding  problems. 


Finally,  we  present  the  modification  of  GLP  and  its  convergence 


analysis  in  Section  4. 


2.  PRELIMIHABIES 


Consider  the  primal  problem: 


f »  =  min  f (x) 


s  .  t .  g  (x)  ^  b  , 


X  e  X , 


where  f  is  a  continuous  real-valued  function,  g  is  a  continuous 


function  from  R"  to  R"  ,  X  is  a  nonempty  compact  subset  of  R",  am 


b  is  a  vector  in  R"*.  The  Lagrangian  dual  of  (PI)  is: 


L*  =  maximize  {  L(u)  :  u  i  0 ,  u  e  R“  ) 


where  L(u)  =  minimum  (f(x)  +  u[g(x)-b]  ;  x  e  X) ,  and  xy  denotes 


the  usual  dot  product  between  vectors  x  and  y. 


When  L(u)  is  relatively  easy  to  evaluate,  the  pair  of 


problems,  (PI)  and  (Dl) ,  can  be  addressed  by  GLP.  Below  we  state 


the  algorithm  in  the  dual  cutting  plane  form  (Zangwill,  1969) . 


Step  0.  Find  a  point  Xo  e  X  such  that  g(xo)  <  b.  Let  k  =  1,  and 


go  to  Step  1 


Solve  the  k-th  master  problem: 


(w,  u) 


s.t.  w  i  f  (Xi)  +  u[g(xi)-b] 


for  i  =  0,...,k-l 


u  i  0 


't;.  >:4*» 


Let  (Wk.Uk)  be  an  optimal  solution  and  go  to  Step  2. 

Step  2.  Solve  the  k-th  subproblem: 

SI:  min  {  f  (x)  +  UkCg(x)-b]  :  x  e  X  } 

Let  Xk  be  an  optimal  point,  and  let  L(Uk)  =  f  (Xk) 
UkCg(Xk)-b].  If  Wk  -  L(Uk)  .  Uk  is  an  optimal  dual 
solution.  Otherwise,  if  Wk  >  L(Uk),  then  replace  k  by 
k  +  1 ,  and  go  to  Step  1 . 

The  k-th  master  problem  (Ml)  is  a  linear  programming  proble 
with  the  following  dual: 

k-1 

M2:  min  £  Xtf  (xj 

n  i  *0 

k-1 

s.t.  Z  ntg(x»)  i  b 
i  =  0 

k-1 

Z  n.  =  1, 
i  =  0 

n,  i  0  for  i  =  0 ....  ,k- 1 

When  (M2)  replaces  (Ml)  in  Step  1,  the  resulting  algorithm 
is  generally  known  as  Dantzig-Wol f e  decomposition  (1960,  1961), 
column  generation  or  GLP .  Qeoffrion  (1970)  also  classifies  the 
algorithm  in  the  cutting  plane  form  as  the  strategy  of  outer 
linearization  and  relaxation,  or,  in  the  column  generating  form 
as  the  strategy  of  inner  linearization  and  restriction. 


3. 


A  CLASS  OF  DIBECTION  FINDING  PBOBLEMS 


Consider  now  a  feasible  direction  scheme  for  solving  (Dl). 

At  a  given  feasible  point  Uic  and  a  parameter  o,  one  possible 
direction  finding  problem  is 

DFl:  Max  [g  (xi«) -b]  (v-uk) 

V 

s . t .  a  i  L (v)  , 

V  i  0. 

where  xic  is  a  solution  of  (SI),  so  that  [g(x>c)-b]  is  a 
subgradient  of  L  at  Uie.  The  first  constraint  of  (DFl)  requires 
that  V  be  in  the  level  set  of  L  as  defined  by  a.  If  a  >  L(u>e)  , 
any  feasible  solution  v  to  (DFl)  would  produce  an  ascend 
direction  of  the  form  v-ux,  and  the  best  value  for  a  is  L» . 

Figure  3.1  illustrates  an  instance  of  (DFl)  in  which  a  is  set  to 
L(ux)  (=  7.0  1  and  the  optimal  solution  of  the  resulting  problem 
is  denoted  by  Vx .  Note  that  the  direction  Vx-ux  is  an  ascent 
direction  for  any  choice  of  subgradients. 

In  practice,  L(v)  must  be  approximated  since  it  is  not 
always  available  in  closed  form.  Each  different  approximation 
would  yield  a  different  direction  finding  problem.  Similarly,  the 
different  choices  for  a  and  Xx  would  also  yield  different 
direction  finding  problems.  To  ensure  that  vx-Ux  is  an  ascent 
direction,  the  approximating  function  along  with  the  value  for  a 
and  the  vector  xx  must  be  carefully  chosen.  Below,  we  examine  two 
different  direction  finding  problems  derived  from  (DFl). 


Uu)  *  5.0 


L(u)  =  6.0 


L(u)  =  7.0 
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Figure  3.1:  An  illustration  of  direction  finding  problem  (DFl) 


First,  we  consider  the  direction  finding  problem  associated 
with  GLP  in  the  cutting  plane  form.  From  the  master  problem  (Ml) , 
it  is  clear  that  the  approximation  of  L  is 


Lai (v)  =  min  {  f(xi)  +  v[g(xi)-b]:  i=0,...,k-l  ). 


Then,  Hearn  and  Lawphongpanich  (1987)  showed  that  Ujc«i,  the 
solution  to  the  (k+l)-st  master  problem,  solves  the  following 
problem  derived  from  (DFl): 


DF2: 


max  [g  (Xje)  -b]  (v-u»e) 


.  t  .  Wie*  1  i  Lai  (v)  , 
V  k  0  , 


where  wk*i,  the  optimal  objective  function  value  of  the  (k+l)-st 
master  problem,  replaces  a.  Define  X(uje)  as  the  set  of  solutions 
to  (SI).  When  X(uj«)  is  a  singleton,  L  is  differentiable  at  uic , 
and  [g(xie)-b]  is  the  only  choice  of  ’subgradient’  for  the 
objective  function  of  {DF2)  .  However,  when  X(ujt)  is  not  a 
singleton,  L  is  nondi f f erentiable  and  the  choice  of  objective 
functions  becomes  infinite.  Moreover,  an  arbitrarily  chosen 
subgradient  could  result  in  (DF2)  generating  a  nonascent 
direction  as  shown  in  Hearn  and  Lawphongpanich  (1907) .  Below,  we 
consider  a  second  direction  finding  problem  in  which  [g(xjt)-b]  is 
replaced  with  a  set  that  includes  the  subdifferential  of  L  at  uk . 


Assume  that  X  can  be  represented  by  a  finite  discrete  set 
tyii-'-tyT).  For  example,  when  (PI)  is  a  bounded  integer  program, 
then  X  is  itself  a  finite  set  of  vectors  with  integral  elements. 


T 
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Then.  X(ui«}  is  a  finite  discrete  set.  For  the  discussion  below, 
define  for  any  u  ^  0 

X»(u)  =  {  yj :  f (yj ) +u (g (yj ) -b)  s  L(u)  +  e ,  yj  e  X  } . 

that  is,  Xc(u)  is  the  set  of  all  e-approximate  solutions  of  (SI). 
When  e  =  0.  Xo(u)  is  simply  the  set  of  solutions  to  (SI)  which  is 
previously  denoted  as  X(u) .  Since  L  is  concave,  the  e- 
subdi f f erential  of  L  is  well  defined  and  can  be  written  as 
follows  (see,  e.g.,  Clark,  1975;  Kiwiel,  1985;  Lemarechel ,  1980; 
and  Zowe .  1987) : 

jcL(u)  =  {  h:  L(v)  i  L(u)  +  h(v-u)  +  e,  V  v  i  0  ) 


Similarly,  when  e  =  0,  <SoL(u)  =  5L(u)  ,  the  subdifferential  of  L 
at  u,  and  3L(u)  is  also  equivalent  to  conv{  [g(yj)-b]:  yj  e 
X(u)},  where  conv{-)  denotes  the  convex  hull  of  a  set.  Below, 
Theorem  3.1  relates  X>(u)  to  the  e-subdif f erential  of  L(u). 


Theorem  3.1:  Conv{  [g(yj)-b]:  yj  e  X«(u)  }  is  a  subset  of  5«L(u). 

Proof :  For  every  v  i  0  and  for  each  yj  e  Xs(u) 

.L(u)  +  [g (yj ) -b ] (v-u)  =  L(u)  -  [g(yj)-b]u  +  [g(yj)-b]v 

i  f{y4)  -  e  +  tg(yj)-b]v 
i  L  (v)  -  e ,  or 

L(u)  +  [g (yj ) -b] (v-u)  +  e  i  L(v),  (3.1) 

where  the  first  inequality  follows  from  the  definition  of  Xe(u) 
and  the  second  inequality  from  the  definition  of  L.  Since 


jr 


equation  (3.1)  holds  for  all  yj  e  Xe(u) ,  it  must  hold  for  all 
convex  combinations  of  yj  e  Xk(u) ,  and  the  theorem  follows.  ^ 

Given  the  above  definitions,  (DFl)  can  be  modified  as 
f  ol lows : 

DF3:  max  min  {  [g  (yj  ) -b ]  (v-uic)  :  yj  e  X«(uic)  } 

s.t.  L(u»e)  i  L>.2(v) 

V  i  0  , 

where 

Las  =  min  {  f (x)  +  v[g(x)-b]:  x  e  U  (xo)  ),  and 

k-1 

=  any  subset  of  U  Xs(ui). 

i  =  l 

If  X(u>c)  replaces  XeCutc)  in  the  objective  function  of  (DF3)  ,  the 
'min’  part  in  the  objective  is  the  expression  for  the  directional 
derivative  of  L  at  Ui«  in  the  direction  (v-u>c)  .  Note  that  (DF3) 
can  be  written  in  the  cutting  plane  format  as  follows; 

DF4:  max  w 

s.t.  w  i  [g  (yj  ) -b]  (v-uic)  V  yj  e  X«{uic) 

L(uj«)  ^  f(x)  +  vCg(x)-b]  V  X  e  U  (xo) 

V  i  0 . 

Under  the  assumption  that  X  is  a  finite  discrete  set,  L(u) 
can  be  equivalently  written  as  a  minimum  of  a  finite  number  of 
linear  functions.  Then,  (DF3)  and  (DF4)  resemble  the  direction 
finding  problem  of  an  algorithm  proposed  by  Demyanov  (see, 
Demyanov  and  Malozemov,  1974,  and  Lemarechel,  1980)  in  that  they 
require  a  full  knowledge  of  the  E-subdifferential.  However, 


Deoiyanov’s  direction  finding  problem  is  nonlinear  whereas  (DF4} 
is  linear. 


As  stated  in  (DF3)  ,  M*®"^  can  be  any  subset  of  the  union  of 
the  previously  calculated  e-subdifferentials.  Theoretically, 
can  be  an  empty  set  and  the  algorithm  to  be  presented  in  the  next 
section  would  still  converge.  However,  taking  M^~’'  as  an  empty 
set  means  that  L(v)  is  approximated  by  only  one  hyperplane 
defined  by  xo .  In  practice,  this  may  not  yield  good  directions. 
Thus,  the  choice  of  the  set  M*®"^  should  allow  a  good 


approximation  of  L(v)  and,  in  turn,  its  contours  in  the 


neighborhood  of  uie. 


4. 


An  Ancant  Algor itha  and  its  Convergence 


The  algorithm  below  is  a  modification  of  the  cutting  plane 
algorithm  in  which  we  replace  the  master  problem  (Ml)  with  the 
direction  finding  problem  (DF3)  [or,  equivalently,  (DF4) ]  and  add 
a  line  search  step. 


An  Ascent  Algorithm 

Step  0.  Let  xo  e  X  satify  g(xo)  <  b.  Select  ui  ^  0  and  compute 
X.(ui).  Set  M<=>  =  0  and  k  =  1. 

Step  1 .  (Direction  Finding)  Solve  problem  (DF3)  (or, 

equivalently,  (DF4) )  and  let  wk  denote  the  value  of  the 
optimal  objective  function,  and  vw  denote  the  solution.  If 
wic  ^  0 ,  stop  and  ux  is  an  optimal  solution.  Otherwise,  go  to 
Step  2. 

Step  2.  (Line  search)  Solve 

L(ux  +  tx(vx-ux))  =  max{  L(ux  t(vx-UK))  :  0  i  t  ^  1  } 

t 

and  set  ux*i  =  ux  tx(vx-Ux)  .  Go  to  Step  3. 

Step  3.  (Evaluate  L(ux->i))  Solve  the  subproblem  (SI)  and 

construct  the  set  X«(ux-i).  Set  k  =  k'fl,  and  go  to  Step  1. 

In  Step  0,  the  point  xo  satifies  the  Slater  constraint 
qualification,  and  by  construction  xo  is  included  in  the 
approximating  function  Lxa .  This  prevents  vx  from  being  unbounded 
since  v[g(xo)-b]  — >  -o#  as  if  any  component  of  v  goes  to  <» .  For 


the  convergence  analysis  below,  it  is  assumed  that  the  set  (u:  a 


£  L(u)}  is  bounded  for  all  a.  Thus,  uie  is  bounded  for  all  k  as 


well. 


The  first  three  theorems  justify  the  direction  finding 


problem  (DF3) . 


uic  is  a  solution  to  (Dl)  if  and  only  if  Uk  solves 


(DF3)  . 


Proof :  For  convenience,  assume  that 


1)  X(uic)  =  {yi . y^Jt 

2)  X«(u}c)  *  {yi ,  .  .  .  ,y^,y».-i . .  •  .  .y<»>  .  and 

3)  M**" *  =  {xi,...,x«}. 


where  p  S  q  and  q  T. 


Let  ujc  be  a  solution  to  (Dl)  .  Then,  there  exists  a  (n,a) 
satisfying  the  following  Karush-Kuhn-Tucker  (KKT)  conditions: 


2  nj[g(yj)-b]  +  2  Ore,.  =  0 
j=l  r=l 

P 

2  xi  =1 

j  =  l 

a»*[uj«]*.  =  0,  r  =  l, 
n  and  o  ^  0 . 


where  e>.  is  the  r-th  unit  vector  in  R"*  and  [z]r  denotes  the  r-th 
component  of  the  vector  z.  Given  this  pair  of  multipliers,  (x,a), 
define  the  triplet  (n’,b’,o’)  as  follows: 


Xj  ®  Xj,  j  =  l,...,p 


0. 


j  =  P+1 . q 


ei=o, 

Or  =  OIr  I  r  =  1 ,  .  .  .  ,m. 

Then,  (n’,&’,a’)  satisfies  the  following  KKT  conditions  for  (DF3) 
at  the  point  uie:. 


q  c  m 

2  nHg(yj)-b]  +  2  6i[g(xi)-b]  +  2  oier  =  0 
j=l  i=l  r=l 

q 

2  nj  =1 

j  =  l 

ei[g(xi)-b]  •  =  0 

oiCujelr  =  0 

n ’ ,  & ’ , and  o’  i  0 . 


Since  uk  is  feasible  to  (DF3)  ,  Ujc  must  be  optimal  to  (DF3)  as 
well. 


Assume  that  u>«  solves  (DF3)  .  Then,  there  exists  a  (n’,b’,o’) 
satisfying  the  above  KKT  conditions  for  (DF3) .  Since  n*  and  B' 
are  nonnegative  and  2jnj  =  1 ,  we  can  define 


a  = 

2jnj  + 

2, 

lBI 

n’j  = 

nj  /a, 

j 

=  1 

»  •  ♦  • 

-P. 

P+1 . 

q 

b;  = 

Bi/a, 

i 

=  1 

f  •  •  • 

,s 

Or  = 

oii/a , 

r 

=  1 

9  «  •  • 

,m. 

Then , 

(n‘  ,B'  ,0< 

’)  is 

an 

optimal 

set 

of 

mu 1 t i p 1 

iers  at  the  point 

(z ,  v) 

=  (L(Ule) 

,  Ui«)  f 

or 

the 

fol 

lowi 

ng 

problem 

which  is  related  to 

(DF4) : 


DF5 :  max  z 


s .  t . 

z  i 

f(yj)  + 

Cg (yj ) -b]v ,  j  =  1 , .  . 

•ipiP+li^-^q 

z  i 

f(Xt)  + 

[g  (xi) -b]v,  i  =  1 ,  .  . 

.  .S 

V  i 

0, 

that  is,  (L(ux) 

.Ux) 

solves 

DF5.  However,  note  that 

(Dl)  can  also 

be  written  as: 

D2 :  max 

z 

s .  t . 

z  a 

L(v) 

V  i 

0. 

Then,  (DF5)  is 

a  relaxation 

of  (D2) .  Since  (L(ux), 

Ux)  solves 

(DF5)  and  is  feasible  (D2) , 

it  must  be  optimal  to 

(D2)  .■ 

Theorem  4.2;  If  ui*  does  not  solve  (Dl)  ,  then 

min(  Cg (y j ) -b3  (vu-uie>  :  yj  e  X«(ui<)  )  >  0, 
where  vic  solves  (DF3)  . 

Proof :  By  Theorem  4.1,  uk  does  not  solve  (Dl)  implies  that  uic  is 
not  a  solution  to  (DF3) .  However,  uk  is  still  feasible  to  (DF3) , 
so  the  following  inequality  must  hold: 

min{  [g  (yj  ) -b]  (vi«-ui*)  :  yj  e  X.fuic)  } 

®  max  {  min(  Cg  (yj ) -b]  (v-uie)  :  yj  e  X«(ui«)  )} 

V  feasible 

>  min{  [g  (yj ) -b]  (ux-uj*)  :  yj  e  X«(ux)  }  =  0.| 

It  should  be  noted  that  the  above  two  results  also  hold  when 


LAa(v)  is  replaced  by  the  actual  Lagr^ngian  function  L  in  (DF3) . 


The  following  series  of  lenunas  and  a  theorem  demonstrate 
that  the  ascent  algorithm  above  converges  to  an  optimal  solution 
of  (Dl) .  Furthermore,  they  are  similar  to  the  standard  argument 
for  establishing  convergence  of  feasible  direction  algorithms  in 
nonlinear  programming. 

Lemma  4.3:  There  exists  an  e  >  0  such  that  if  uje  — >  u«  ,  then 
X.Cujc)  =  X(u»)  ,  for  k  sufficiently  large. 

Proof  :  Let  X{u»)  =  {yi . •  Define 

i)  hi(u)  =  max  {f(yj)  +  u[g(yj)-b]:  j  =  1 . p) 

ii)  haCu)  =  min  {f(yj)  +  u[g(yj)-b]:  j  =  p+1 . T> 

iii)  a  =  [hi (u«) +h2 (u») ]/2 

At  u» ,  the  following  hold: 

A)  hi(u»)  =  L(u»)  =  f(yj)  +  u»[g(yj)-b],  j  =  1 . p 

B)  hi(u»)  <  a  <  ha(u*) 

C)  <7=  [a  -  hi  (u»)  ]  >  0 


From  the  continuity  of  hi(u) ,  haCu) ,  and  L(u) ,  there  must  exist 
an  integer  K  sufficiently  large  so  that  for  every  k  2:  K 


D) 

1  h  1  (  uk  ) 

-  hi (u») 

1  <  <7/2 

E) 

1 L ( Uk ) 

-L(u«) 1  < 

<7/4 

F) 

f  or  j  = 

1 . P 

If  (y:. 

)  +uj.[g  (yj 

)-b]  - 

=  I  f  (yj  ) +uj,[g  (yj  ) -b]  -  f  (yj  ) -u*  [g  (yj  ) -b]  I 
<  <7/4. 


s 


1 


1 


I 

! 


J 


Q)  ha(uic)  >  a. 


From  (E)  and  (F)  ,  we  have  that  for  J  =  1 . p 

I 

! 

I  f  (yj  )  +Ule[g  (yj  )  -b]  -L(u»e)  I  i 

( 

s  I  f  (yi ) +uieCg  (yj ) -b3  -  L<u«)  |  +  |L(u«)  -  L(ui.)|  _  ' 

i  0/2,  j 

and  since  L(ui«)  i  f  (yj )  +u»e[g  (yj ) -bJ ,  V  J,  it  follows  that 

I  f  (yj  ) +ui«[g  (yj  ) -b]  -L(uic)  I  =  f  (yj  ) +uje  tg  (yj  ) -b  ]  -L(uj.)  i  <7/2,  or 
f  (yj  )  ♦uje[g  (yj  ) -b]  i  L(uie)+<7/2. 

Thus,  for  any  e  between  (l/2)<7  and  (3/4)  <7, 

f(yj)  +  ui«[g(yj)-b]  i  L(U}c)  +  e,  for  j  =  1 . p,  (4.1) 

that  is,  yj  is  an  element  of  X>(uic)  for  j  =  l,...,p.  Moreover, 
from  (E) , 

L(u>c)  i  L(u»)  +  <7/4 

L(uic)  e  i  L(u»)  ♦  <7/4  +  s 
S  L(u»)  ♦  o 
=  a 

<  f  (yj  ) +ui«[g  (yj  ) -b]  ,  for  j  =  p+l,...,T.  (4.2) 

where  the  third  inequality  follows  the  above  selection  of  e,  the 
equality  from  (C) ,  and  the  last  inequality  from  (G)  and  the 
definition  of  ha(u).  This  means  that  yj  does  not  belong  to 
X»(uic),  for  j  =  p+l,...,T.  Therefore,  Xe(uic)  =  X(u»).  || 


i  4.4:  For  a  given  direction  d, 
mini  g'd:  g’  e  conv ( tg (yj ) -b] :  j  =  ) 

=  mini  [g(yj)-b]d:  j  =  1 . p  } 


Proof :  The  result  follows  from  the  fact  that  the  problem  on  the 
left  hand  side  can  be  stated  as  a  linear  program 


min  Z  ( [g  (yj  ) -b  ]d)  nj 
j*l 

P 

s . t .  Z  nj  =  1 

1  =  1 


nj  i  0.  j  = 


which  always  yields  an  extreme  point  solution.  | 


t  4.5:  If  X.(uie)  =  {yi,...,y^}  and  mini  [g(yj)-b]d:  j  = 
l,...,p  )  >  0,  then  there  exists  a  t  >  0  such  that 

L(uie+<Td)  i  L(uj,)+amini  [g(yj)-b]d:  j  =  l,...,p  } 
forallOioiT. 


Proof :  Assume  without  loss  of  generality  that 


0  <  [g(yi)-b]d  =  mini  [g(yj)-b]d:  j  =  1 . p  } 


Define 


“T  =  min 


i  f  (yj  ) +u[g  (yj  ) -b  ]  -  L(ui«)}  for  all  j  such  that 

- .g(yi)d  >  g(yj)d  &  j  ^  X.iuic) 

[g(yi)  -  giyj) ]d 


and  observe  that  nr  >  0  since 


f(yj)  +  utg(yj)-b]  >  L(ui«) 


for  j  t  X«  (u>*)  , 


and  [g (y i) -g (yj ) ]d  >  0  by  construction.  Thus,  for  any  0  i  o  S  t 
and  avary  j  f  X>(u}c)  and  g(yi)d  >  g(y4)d 


{f<yj)  +  uieCg(yj)-b]  -  L(uie) 
<T  S  - 

[g (yi)  -  g (yj )  ]d 


a[g(yi)-b  -  g(yj)+b]d 

f  lyj ) 

♦  UieCg(yi)-b]  -  L(u>e) 

L(uic)  +  o'[g(yi)-b]d 

f  (yi) 

♦  (uic+cd)  [g  (Ui  ) -bl 

(4.3) 

However,  for  J  g  X«(uk) 

and  0  < 

[g(yi)-b]d  i  [g(yi)-b]d 

L(uic) 

< 

f  (yi) 

+  uie[g(yi)-b] 

L(Ufc)  +  <TCg(yi)-b]d 

< 

f  (yi) 

+  (ui«+<Td)  [g  (yi  )  -b] 

(4.4) 

and  for  J  e  X«(uic) 

L(Uk) 

f  (yi) 

♦  Ui«[g(yi)-b] 

L(uie)  +  <r[g(yi)-b]d 

f  (yi) 

+  (Ufc-KTd)  [g  (yi  )  -b) . 

(4.5) 

Combining  (4.3),  (4.4)  and  (4.5),  we  have  that  for  0  i  (7  i  y 

L(u>e)  +  o[g(yi)-b]d  i  min{  f  (yj  )  ♦  (u>,+<7d)  [g  (yj  ) -b  ] :  j  =  l,...,T) 

=  L  (uic+ad)  .  p 

Lemma  4.6:  Assume  that  e  is  chosen  as  in  Lemma  4.3  and  the 

algorithm  generates  a  sequence  (um) .  Then,  there  cannot  be  a 
subsequence  (uic)  ,  k  e  0,  with  the  following  properties: 


i) 

Uk  — > 

u. , 

k  e 

ii) 

Vi,  — > 

V-  , 

k  e 

Q ,  and 

iii) 

min(  g 

■  (V--U-) ;  g* 

e  conv ( C g (y ) -b ] :  y  e  X(u»))  }  >  0, 

'4 

■i 

I 


I 


i 


where  0  is  a  subset  of  (1,2,3,...). 


Proof :  Assume  that  X(u.) 


(1 . p)  .  By  (ill)  and  Lemma  4.4, 


t  . 


i 


min(  g*(v«-u-):  g'  e  conv ( Cg (y ) -b ] :  y  e  X(u«))  } 

=  min  {  [g (yj ) -b] (v*-u«)  :  j  =  1 . p  }  =  B  >  0. 

From  Lemma  4.3,  there  must  exist  a  sufficiently  large  such 

that  Xe(uic)  =  (1 . p)  for  some  k  ^  and  k  e  n.  Since  the  set 

(l,...p}  is  finite,  there  must  exist  a  index  j*  such  that 

=  arg  min{  [  g  (y  j  ) -b  ]  (vie-ui,)  :  j  =  1 . p  }  (4.6) 

infinitely  often.  Define  to  be  the  subset  of  0  for  which  j»  is 
the  index  which  yields  the  minimum  value  for  the  right  hand  side 
of  (4.6).  For  convenience,  we  assume  that  =  1.  Then,  we  have 
that 


Lim  [g (yi) -b] (vk-uk)  =  [ g (y i ) -b ] ( v«-u«)  =  6  >  0, 

k  e 

and  it  follows  that  there  exists  K“  i  such  that 

[  g  (y  1 ) -b  ]  ( Vi«-u»e)  >  6/2  for  k  i  K®  and  k  e  f)^.  (4.7) 

Thus,  at  uie  the  direction  (vk-uj<)  is  an  ascent  direction. 
Moreover,  since  u»c*i  maximizes  L(u)  along  the  direction  (vic-u>c)  , 


L(uic«i)  i  L(uj«  +  a(vj*-Uj«))  for  0  <  a  <  t  and  k  e  (1^ 

i  L(uie}  +  <T [ g  (y  1 ) -b ]  ( vi«-u>« )  for  0  <  ct  <  t  and  k  e  (1’ 
i  L(u>c)  +  (<tB) /2  for  0  <  (7  <  T  and  k  e  0^,  (4.8) 


where  nr  is  as  defined  in  Lemma  4.5  and  the  last  two  inequalities 
follow  from  Lemma  4.5  and  equ.(4.7).  Letting  k  approach  infinity 
on  the  subsequence  ,  (4.8)  yields 


mmmm 


\ijfv  m  u."  v*  v’.H?.'ir<'y.r*rs' 


L(u-)  i  L(u.)  +  (<Te)/2 

which  is  a  contradiction  since  both  <t  and  &  are  positive. || 

Theorem  4.7:  Assume  that  e  is  chosen  as  in  Lemma  4.3.  If  the 

algorithm  generates  a  sequence  (uie)  ,  then  it  must  converge 
to  an  optimal  solution  of  (Dl) . 

Proof :  Assume  that  the  algorithm  generates  an  infinite  sequence 
of  points  and  none  of  which  is  a  solution  to  (Dl)  .  Since  (uic) 
lies  in  a  bounded  region,  there  must  exist  a  convergent 
subsequence,  i.e.,  uie  — >  u. ,  for  k  e  (1,  where  D  is  a  subset  of 
(1,2,3,...)  and  u*.  does  not  solve  (Dl)  . 

Assume  that  X(u.)  =  (l,...,p).  By  Lemma  4.3,  Xe(uic)  -  X(u.) 
for  k  sufficiently  large,  i.e.,  k  i.  Kx ,  thus  Xe(u«)  =  X(u-). 
Since  u*  is  not  a  solution  to  (Dl) , 


0  <  3*  =  max  min{  [g (yj ) -b] (z-u«) : 
z  j 

s.t.  L(u«)  ^L(z) 

z  i.  0 . 


1 . P  > 


For  k  i  Ki ,  def ine 


4 

'j 

N 

I 


DF6:  Bi«  =  max  min{  C  g  (y^ ) -b  3  (z-Uk)  :  j  =  l,...,p  } 

z  J 

s.t.  L(ui«)  ^L(z) 

z  i  0 . 

Note  that  3i«  varies  continuously  with  u>c.  Thus,  Bic  — >  B»  for  k  e 
0  and  k  2  Ki . 


M 

k 

1 

7", 

>. 


Now,  let  Vk  be  a  solution  to  (DF3)  defined  at  uic.  Then,  by 


construction,  vie  is  bounded  and  there  must  exist  a  subset,  ,  of 
Q  such  that  vj,  — >  v«  on  Moreover,  since  {DF3)  is  a 

relaxation  of  (DF6) , 

&JC  i  min{  [g  (yj ) -b]  (vje-uje)  :  j  =  l,...,p  },  (4.9) 

for  k  i  Ki  and  k  e  {)’•.  From  the  finiteness  of  of  the  set 
{l,...,p},  there  must  exist  an  integer  such  that 

j*  =  arg  min{  C  g  (yj  ) -b  ]  (vje-ujc)  :  j  =  l,...,p  )  (4.10) 

infinitely  often.  Let  Q®  be  a  subset  of  for  which  is  the 
index  which  minimizes  the  right  hand  side  of  (4.10)  and  for 
convenience  assume  that  j»  =  1.  Then,  combining  (4.9)  and  (4.10) 
gives 


[g  (yi) -b]  (vi«-uk)  i  Sic  for  k  i  Ki  and  (1“  . 

Taking  the  limit  on  both  side  with  respect  to  (1®,  we  have  that 
lim  [g  (yi) -b]  (vic-uj«)  =  [g  (y  i ) -b  ]  ( v«-u-)  i  &•  >  0 
which  contradicts  Lemma  4.6.  ^ 

Therefore,  if  the  algorithm  terminates  after  a  finite 
number  of  iterations.  Theorem  4.1  and  4.2  guarantee  that  uie 
solves  (Dl).  Otherwise,  the  algorithm  generates  an  infinite 
sequence  which,  by  Theorem  4.7,  converges  to  an  optimal  solution 
of  (Dl).  Also,  it  is  interesting  to  note  that  although  the 
algorithm  uses  e-subdifferential  in  calculating  its  ascent 
directions  an  exact  optimal  solution  can  be  obtained  by  choosing 


e  correctly.  In  general,  one  expects  algorithms  using  an  e- 
subdi f f erential  to  produce  e-optimal  solutions,  possibly  in  a 
finite  number  of  iterations.  We  consider  this  type  of  algorithms 


in  a  separate  study  to  appear  later. 
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